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Methodology 


From July 15 to October 13, 2015, Hart Research conducted an online survey 
among 325 Chief Academic Officers or designated representatives at AAC&U 
member institutions to measure the prevalence of specified learning outcomes in 
higher education institutions today and to document priorities and trends in 
undergraduate education. The margin of error is +4.4 percentage points for the 
entire sample, and it is larger for subgroups. The total population for the survey 
included 1,001 AAC&U member institutions that were invited to complete the 
survey, and thus the response rate for the survey is 32%. The sample is 
representative of AAC&U’s total membership in terms of both institution type (11% 
associates, 30% bachelor’s, 39% master’s, 19% doctoral/research, and 1% other) 
and affiliation (46% public, 28% independent, 25% religious, and 1% proprietary). 


Additionally, from August 4 to September 24, 2015, Hart Research conducted 14 in- 
depth interviews (IDIs) with higher education leaders who completed the online 
survey. Individuals who indicated in the survey that their institution is tracking and 
disaggregating data and setting goals related to diversity and equity were invited to 
participate in these in-depth interviews. Respondents represented institutions from 
various states, Carnegie classifications, and included some _ minority-serving 
institutions. The interviews were conducted by telephone and lasted approximately 
45 minutes to one hour. The discussions were designed to provide a deeper 
understanding of how diverse institutions are focusing on advancing equity and 
closing attainment and achievement gaps. 


Select findings from the survey and in-depth interviews about priorities for tracking 
and advancing the success of underserved students are outlined in this report. 
Forthcoming reports will provide analysis of findings related to AAC&U’s member 
institutions’ reporting on general education design elements and assessment of 
student learning. 
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Overview 


= Most AAC&U member institutions have a common set of learning 
outcomes that apply to all students, and their overarching focus is on 
making changes to advance success for all students. Many institutions 
are tracking students’ achievement of learning outcomes, but few are 
disaggregating this data by various student characteristics (e.g., 
race/ethnicity, income levels, or parental education levels). 


= AAC&U member institutions are implementing a variety of evidence- 
based interventions to advance all students’ success with a focus on 
those that support the successful transition to college. These include 
several high-impact educational practices. 


« Many AAC&U member institutions are tracking and disaggregating data 
on the retention and graduation rates of students from traditionally 
underserved groups. Far fewer institutions are disaggregating data on 
students’ participation in high-impact learning practices or on their 
achievement of institutional learning outcomes. 


= AAC&U member institutions are trying to advance success among 
students who traditionally have been underserved by higher education, 
but they are doing so primarily through strategies embedded within 
their broader efforts to advance success for all students. Their 
strategies for closing equity gaps are still works-in-progress that they 
are continuing to expand and refine. 


«= Some AAC&U member institutions are setting explicit goals for closing 
gaps in retention and/or on-time graduation rates for students from 
different racial and ethnic groups. Far fewer are setting explicit goals 
for closing achievement gaps in student learning outcomes or 
participation in high-impact educational practices. 


« Many AAC&U member institutions are taking proactive steps to build 
faculty and staff capacity to use culturally competent teaching 
strategies and are working to expand opportunities for high-impact 
learning, especially among first-generation, low-income _ students, 
and/or students of color. 
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Key Findings 


Most AAC&U member institutions have a common set of learning 
outcomes for all of their students. 


Fully 85% of AAC&U member institutions report that they have a common set of 
intended learning outcomes for a// undergraduate students. This is up from 78% of 
institutions that said they had common learning outcomes in 2008. In addition, 
70% of institutions are tracking students’ achievement of these learning outcomes. 


Does your institution have a common set of intended 
learning goals or learning outcomes that apply to ALL 
undergraduate students? 


u Yes, have common set of intended learning goals/outcomes No, do not have 


85% 
78% 


November/December 2008 July — October 2015 


Many institutions are implementing evidence-based practices to 
advance student success, and they are most likely to require high- 
impact interventions that support the successful transition to college. 


In the in-depth interviews, administrators discuss the value of high-impact learning 
practices both in supporting higher rates of persistence and higher levels of 
achievement of learning outcomes. Some also note that student demand for these 
experiences is high. 


Of the 10 high-impact interventions tested in the survey, however, there are only 
three that a majority of institutions require for all their students, and two of those 
focus on supporting first-year students. Fully 60% of AAC&U member institutions 
indicate that their campus requires a// students to engage in first-year experiences 
that support the transition to college, and more than half (52%) require a// students 
to participate in first-year academic seminars. Additionally, a littke more than half 
(52%) require a// students to take global or world culture studies. 


Notably fewer (34%) require a// students to participate in diversity studies and 
experiences. Other evidence-based, high-impact practices such as service learning, 
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learning communities, undergraduate research, and internships are offered by 
many institutions, but few require all students to participate in them. 


Required 
of All 
Students Optional 
% % 
First-year experiences that support the transition to college 60 chil 
First-year academic seminars 52 30 
Global or world culture studies 52 41 
Diversity studies and experiences 34 53 
Service learning in courses 14 79 
Learning communities 12 59 
Undergraduate research 9 87 
Practicums and supervised fieldwork 7 90 
Internships 6 92 
Study abroad 2 94 


Many AAC&U member institutions are tracking and disaggregating data 

on the retention and graduation rates of students from traditionally 
underserved groups. Far fewer institutions are disaggregating data on 
participation in high-impact learning practices or on achievement of 
institutional learning outcomes. To the extent that institutions are 
disaggregating data, many more are looking at differences by students’ 
race and ethnicity than their socioeconomic status or their parents’ level of 
educational attainment. 


Data tracking on retention and graduation rates is universal across AAC&U member 
institutions, and large majorities of institutions say they track data on participation 
in key high-impact learning experiences (78%), achievement of credit/course 
completion milestones (75%), and achievement of institutional learning outcomes 
(70%). Fewer campuses, but majorities nonetheless, also track enrollment in 
remedial courses (63%) and successful completion of those courses (61%). 
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Institutions report on their tracking and disaggregation of 
data on student persistence, graduation, and/or achieve- 
ment of learning outcomes. 


ulnstitution tracks these data 
= Institution disaggregates these data by race/ethnicity, SES, and/or parents' education 


Graduation rates 100% 
Retention rates 100% 


Participation in high- . 
impact practices EAE MI) 78° 
sn inilostones Mn a 752 
completion milestones EY 75% 
Achievement of a 
learning outcomes 17% 70% 
Enrollment in remedial 5 63% 
courses 33% 
rec a eee 61% 
courses 32% 


More than four in five campuses disaggregate data on retention and graduation 
rates by at least one of three variables (race/ethnicity, socioeconomic status’, and 
parents’ level of educational attainment’). No more than one in three 
disaggregates data in any of the other three areas, however. For instance, while 
78% of institutions say they track participation in high-impact practices, only 31% 
disaggregate that data by at least one of the three variables. Specifically, 30% of 
campuses say they disaggregate participation by race and ethnicity, 16% 
disaggregate by students’ socioeconomic status, and only 12% disaggregate by 
parents’ level of educational attainment. Similarly, while 70% of institutions track 
achievement of institutional learning outcomes, just 17% say they disaggregate 
that data. Only 16% disaggregate by race and ethnicity, 9% by socioeconomic 
status, and a mere 6% do so by parents’ level of educational attainment. 


As illustrated in the accompanying table, in each case, campuses are more likely to 
say they disaggregate data by students’ race and ethnicity than by their 
socioeconomic status or parents’ level of educational attainment. 


1 Tn the in-depth interviews, administrators indicate that looking at Pell-eligible students vs. those who 
are not Pell-eligible is a proxy that many institutions use for disaggregating by socioeconomic status. 


? In the in-depth interviews, respondents indicate that disaggregation by parents’ level of educational 
attainment typically involves looking at data among first-generation students. 
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Institutions’ Tracking and Disaggregation of Data in Key Areas 


Track retention rates 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 
Track graduation rates 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 
Track participation in high-impact learning experiences 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 
Track achievement of credit/course completion milestones 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 
Track achievement of institutional learning outcomes 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 
Track enrollment in remedial education courses 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 
Track successful completion of remedial education courses 

Disaggregate by race/ethnicity 

Disaggregate by socioeconomic status 

Disaggregate by parents’ level of educational attainment 


All 
Respondents 
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“We're a PeopleSoft Campus, and so we keep track of most of those variables 
just through the application process. But then, many of the others, we keep 
track of for a number of reasons. We are an urban campus, first-generation 
campus. We went through a massive general education reform, which I led, 
and it’s probably why I'm sitting in the office I am now, because it was a 
SUCCESS. 

We wanted to be able to look at our students in a disaggregated way and 
target how we wanted to work with the students and intervene where we 
needed to intervene. And so we started disaggregating and tracking our 
students about...four or five years ago. 

So you add up our major endeavor at reforming our gen ed, and making it 
learning outcomes based. You just look at our changing demographics and 
then add on top of that our state’s shift in terms of following Complete 
College America. But in all honesty, also we changed our whole funding 
formula. And so we needed to follow some of these things as well, so we 
could have predictive analytics in terms of what our upcoming funding is 
going to be. 

All of that led to a change in how we collect data.” 

— Public, Doctoral/Research 


“..50% of our students are minority..We have looked at the disaggregated 
data; we know what our Latino graduation rate is, our African-American 
graduation rate, etc., etc., male, fernale, first generation. And so within that 
overall increase, we have established goals within those sub-criteria to get us 
to where we want to go.” 


— Public, Doctoral/Research 


The in-depth interviews indicate that many of the ways in which 

AAC&U member institutions are trying to advance success among 
underserved students are embedded within broader efforts to advance 
success for al// students through the implementation of high-impact 
interventions as described above. Many institutions are _ piloting 
approaches, and their discussions suggest that their strategies for 
advancing student success are works in progress that they are continuing 
to expand and refine. 


Across the in-depth interviews, when administrators are asked about the process 
their institutions are using to advance the success of underserved students, they 
describe varied approaches that their campuses have undertaken to improve the 
success of a// students. While some programs and efforts are aimed specifically at 
providing resources or support to underserved students (discussed later in this 
report), their focus largely is on furthering the success of a// students. 


In discussing efforts to advance student success on their campuses (including that 
of underserved students), to a large extent administrators focus on the efforts 
aimed at increasing retention and completion rates. They outline many different 
high-impact interventions that they have undertaken or are beginning on their 
campuses, with a particular focus on reshaping students’ transition to college. 
These interventions fall into four main categories: a) the redesign of developmental 
education, b) the adoption of first-year experiences that are required for all 
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students, c) the transition to early alert systems and intrusive advising, and d) the 
expansion of high-impact learning practices (most of which are optional at this 
point). 


a) Redesigning developmental education: Administrators describe a variety of 
ways in which their campuses are redesigning developmental education to advance 
student success. With underserved students overrepresented in remedial 
education, they believe that these efforts benefit many underserved students with 
improved retention and learning quality. Administrators describe a wide variety of 
changes their campuses are making to redesign remedial education beyond the 
specific strategies described below. 


> Replaced developmental education with a program in which students who 
normally would need remediation are required to take a two-week session prior 
to the start of the semester that provides intensive writing and mathematics 
instruction at no extra charge. This is supplemented by another program, which 
continues into the semester and provides intensive tutoring for students who did 
not achieve everything needed in the pre-semester session. 


> Created a free program in the summer with academic workshops in several 
areas. Students take an evaluative test after the program and can place out of 
remedial classes. 


> Redesigned the approach to remedial math in various ways, including on-time 
remediation, a math bridge program, and modularized instruction. 


> Redesigned the approach to remedial English by separating English language 
learners into an English Language Learner Gateway English course and providing 
a two-semester gateway English course for remaining students legitimately in 
need of remedial English. 


b) First-Year Experiences: Administrators describe these initiatives as 
comprehensive approaches to the first-year experience that link curriculum, early 
alert systems, advising, academic supports, and learning communities. Campuses 
that have introduced these programs or plan to do so describe many similar 
elements and supports. Key components or structures that they describe in the 
interviews are outlined below. 


> Every student is in a cohort of 50 students and takes two courses with that 
cohort—a writing course and an academic inquiry course that introduces them to 
the modes of inquiry in the humanities, social sciences, and natural sciences. 
Every cohort has a peer mentor trained to be a source of support. 


> There is a student success team for each cohort of about 60 to 70 students that 
includes a peer mentor, math tutor, writing tutor (embedded into a class), 
library liaison, technology liaison, and counselor. This team works with the 
faculty. 


> The first-year experience includes a year-long seminar focused on student 
success, including academic planning, career planning, and getting them 
officially into a major by the end of their first year. Seminar instructors are 
students’ first-year advisors. 
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> There are learning communities in the first year—which include the first-year 
seminar and two other classes—so that a student is in at least three classes with 
the same group of peers to provide a sense of community that enhances 
learning. 


> There is a vertically integrated Gen Ed curriculum that includes the first-year 
seminar, second-year seminar, and a capstone course. Learning outcomes are 
introduced in the first year and reinforced in the second, and it includes an 
assessment protocol. 


c) Early Alert Systems and Intrusive Advising: Many campuses are using 
predictive modeling and early alert systems to trigger intervention by an advisor. 
These systems typically are within the structure of the first-year experiences at 
institutions that have such a program in place. 


Some administrators mention that they are using specific retention and completion 
tools that have early warnings. These predictive analytics highlight where supports 
are needed and inform the outreach and support provided by advisors. Some 
institutions also have user-friendly degree tracking tools for faculty and students to 
track a student’s progress. 


The full utility and value of these systems has yet to be realized as their use still is 
in the early stages (or has not yet begun) on most campuses. Some administrators 
note that predictive analytics and early alerts also could be used to tease out data 
on specific groups (e.g., minorities, low-income, or first-generation students) to 
inform interventions for those groups. 


d) Evidence-based, High-impact Practices: As noted above, administrators 
acutely recognize the value of high-impact learning practices. Some institutions 
require high-impact practices through their General Education program, while some 
include a component in their required first-year experience. Many are expanding 
optional high-impact practices such as service learning, internships, study abroad, 
civic engagement, undergraduate research, capstone projects, a campus-wide 
research day, or an introduction to composition class that requires all students to 
present their work. Few require all students to participate in these types of high- 
impact practices beyond what they require through first-year experiences, however. 


e) Other Interventions to Advance Student Success: Colleges and universities 
are doing many things to advance student success that are in addition, and often 
complementary, to the four major areas of focus outlined above. These various 
initiatives and programs include outreach to and transition-coaching for local high 
school students, aggressive onboarding during required new student orientation, 
promoting summer school, ensuring small class sizes, moving to professional 
advisors, adding peer mentors, and ensuring that schedules and course offerings 
align with students’ needs. 


Other administrators describe how their campus has introduced a campus narrative 
about graduation that did not exist before to influence the mindset and the 
conversation, as well as the creation of a time-to-completion program that 
encourages students to complete their undergraduate education in four years and 
makes them aware of the costs if they do not. These additional interventions are 
focused almost exclusively on improving persistence to degree. 
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More than half of AAC&U member institutions have equity goals to 

close racial and/or ethnic gaps in retention and on-time graduation. 
Far fewer have goals to address inequities in achievement of learning 
outcomes or participation in high-impact practices. 


Within each goal, AAC&U member institutions are more likely to have a goal to 
close racial/ethnic gaps than to close gaps by socioeconomic status or parents’ level 
of educational attainment. 


Proportion of Institutions that Have Set Equity Goals for Specific Groups 


All 
Respondents 
% 
My institution has set goals to close gaps in retention and/or on- 
time graduation among students... 
From different racial and ethnic groups Bi 
From different socioeconomic groups 36 
Whose parents have different levels of educational attainment 27 
My institution has set goals to close gaps in achievement of 
student learning outcomes among students... 
From different racial and ethnic groups cul 
From different socioeconomic groups 24 
Whose parents have different levels of educational attainment 14 
My institution has set goals to close gaps in participation in key 
high-impact learning practices among students... 
From different racial and ethnic groups 28 
From different socioeconomic groups 23 
Whose parents have different levels of educational attainment 15 


“..When I talk about closing gaps, I may mean that differently than other 
respondents, because some respondents may be talking about gaps within 
their university, whereas I'm more thinking about the fact that the vast 
majority of our students are low-income...More than 70% are ethnic minorities. 
What I'm trying to do is say as an institution, we want to provide, we want our 
retention and graduation rates, we want the access to high-impact 
practices...We want to be able to close the gap between what is available to 
students at more affluent institutions and more affluent students with ours. 

.. The research on high-impact practices shows that, one, they do have a 
high impact, but they're much more available at more affluent institutions and 
institutions that serve wealthier groups of students, and that's the gap I'm 
talking about...” 


— Public, Master’s 
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Many AAC&U member institutions are taking proactive steps to build 

faculty and staff capacity to use culturally competent teaching 
strategies and they are working to expand opportunities for high-impact 
learning among first-generation, low-income students, and/or students of 
color. 


While only one in three AAC&U member institutions says that they have specific, 
explicit equity goals that are aimed at building new opportunities for high-impact 
learning for first-generation students, low-income students, and/or students of 
color, another 37% say they are planning to develop them. This leaves only 30% 
who do not plan to do so. 


Does your institution have specific, explicit equity goals 
aimed at building new opportunities for high-impact 
learning for first-generation students, low-income students, 
and/or students of color? 


Do not have and do not 


have plans to develop Have equity goals 


30% 33% 


37% 


Do not have but are planning to 
develop equity goals 


Additionally, more than two in five (42%) member institutions have a program to 
build faculty, instructor, and staff capacity to use culturally competent teaching and 
program strategies and be more successful among underserved groups, and 
another 35% say they are planning to develop such a program. That leaves only 
23% who have neither a program nor plans to create one. 
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Does your institution have a program to build faculty, 
instructor, staff capacity to use culturally competent 
teaching/program strategies and/or to be more successful 
with minority, low-income, and/or first-generation college 
students? 


Have program to be more 
successful with minority, low- 
income, first-generation 
college students 


42% 


Do not have and do not 
have plans to develop 


Do not have but are planning to 
develop program 


In the in-depth interviews, many administrators indicate that a key focus of their 
campus’s faculty development efforts relating to diversity and inclusiveness is a 
strong emphasis on hiring and retaining faculty that reflect the student body. They 
are focused particularly on efforts to hire more minority faculty, but they also talk 
about the need for faculty who can relate to first-generation students, low-income 
students, international students, and LGBT students. 


Faculty and staff development programs on topics such as teaching first-generation 
students, working with diverse populations, and developing culturally responsive 
pedagogy vary across the _ institutions, with some programs being more 
comprehensive and robust than others. The central focus is on helping faculty and 
staff recognize cultural differences and understand how to best approach and teach 
students of different backgrounds. Administrators outline a variety of initiatives 
that their campuses have to facilitate greater cultural competence among faculty. 
Below are just some examples of the efforts institutions are undertaking: 


> Providing unintentional bias training to all faculty 


> Embedding a focus on pedagogy, diversity, cultural competency, and inclusive 
classrooms in required new faculty orientations, new faculty experiences, faculty 
learning communities, and general faculty retreats 


> Incorporating a focus on multicultural awareness, unconscious bias, and 
inclusive classrooms into faculty training on high-impact practices 


> Creating a formal setting for students of different backgrounds, 
races/ethnicities, socioeconomic statuses, or sexual orientations to tell faculty 
about their experiences in the classroom 


> Creating cohorts of new faculty that attend an orientation and multiple 
meetings, some of which address topics related to diversity and inclusiveness 
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> Closely linking the Provost’s Office with the Office of Diversity 


While some of the training forums for faculty are required (particularly for new 
faculty), most administrators say their institutions have developed programs and 
forums that are voluntary for faculty, and they stress the importance of getting 
faculty buy-in and engagement with these efforts to ensure their success. 


“And part of what we're trying to do, and this is more of a communication 
strategy than it is any sort of initiative or set strategy, is to sensitize people 
to the notion that different groups do respond differently to failure in classes 
or failure on a test. 

And, for example, this is something I've picked up from our faculty 
experts, is that a Latino student who fails their first exam in a subject doesn't 
respond by saying, ‘well, I guess I didn't study right, I need to go talk to the 
instructor and find out how to study better for their tests,’ but are more likely 
to say, ‘I'm not sure I belong in college, maybe I should drop out because I 
don't really belong here.’ And so it's understanding those kinds of differences 
that we would like for more of our instructors to be tuned in to, so that they 
are culturally sensitive to how they give feedback to students.” 


— Public, Doctoral/Research 


“To bring a talented group of students in is a little hard, but it’s not very hard. 
What’s hard is building an inclusive community. And we’re struggling with 
that..A number of the faculty have been here a long time. There are 
insensitivities, there are unintended biases, there are remarks in class. And 


some [faculty members] say they’re not comfortable in talking about issues of 
race,” 


— Religious, Doctoral/Research 


"So we brought, what historically in [ours] and many institutions, was an 
office that was isolated and dealt with more student affairs and grievances 
and whatnot, and it’s now seamlessly back and forth between Academic 
Affairs and the Provost’s Office and Office of Diversity Initiatives, and that is a 
huge step forward. Because now, it’s just expected that diversity and 
academics go hand in hand.” 


— Public, Doctoral/Research 


Page 13 


